Using Lip Reading Recognition to Predict Daily Mandarin Conversation

Audio-based automatic speech recognition as a hearing aid is susceptible to background noise and overlapping speeches. Consequently, audio-visual has been developed complement the audio input with additional visual information. However, huge improvement of neural networks in task resulted robust reliable lip reading framework that can recognize from alone. In this work, we propose model predict daily Mandarin conversation collect new Daily Conversation Lip Reading (DMCLR) dataset, consisting 1,000 videos 100 conversations spoken by ten speakers. Our consists spatiotemporal convolution layer, SE-ResNet-18 network, back-end module bi-directional gated recurrent unit (Bi-GRU), 1D convolution, fully-connected layers. This able reach 94.2% accuracy DMCLR dataset. Such performance makes it possible for applications be practical real life. Additionally, are achieve 86.6% 57.2% on Wild (LRW) LRW-1000 (Mandarin), respectively. The results show our method achieves state-of-the-art these two challenging datasets.